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Background/context: 

Description of prior research and/or its intellectual context and/or its policy context. 

The Numbers Plus preschool mathematics curriculum supports instruction throughout the 
day, with a primary focus on small group time activities. It is organized according to the five 
content areas of the Principles and Standards for School Mathematics (National Council of 
Teachers of Mathematics: NCTM, 2000): number and operations, geometry, measurement, 
algebra, and data analysis. In keeping with recommendations in Curriculum Focal Points for 
Prekindergarten Through Grade Eight Mathematics (NCTM, 2006), Numbers Plus places more 
emphasis (close to 75% of its activities) on the first three content areas. 

The purpose of the Numbers Plus project was to develop, implement, and test an early 
mathematics curriculum. As a part of that process, the pressing need to clearly define and 
measure implementation fidelity not just in this project but in general when evaluating 
interventions became apparent. 

Fidelity of implementation (FOI) is defined as a degree to which a program is implemented 
as defined by program developers (Dane & Schneider, 1998). Knowing and assessing FOI has 
implications for internal and external validity. Understanding critical features of the 
intervention, determining what program components were or were not implemented, how their 
implementation is related to program effectiveness, and identifying conditions under which the 
implementation could be carried out at a high level are all linked to the importance of assessing 
FOI for understanding of the program, estimating its effectiveness, replication, and scale up 
efforts. 

Dane and Schneider (1998) analyzed 162 studies indicating that only 39 of them reported 
some evidence of assessing FOI. They developed a useful framework by presenting five aspects 
of fidelity: 1) Adherence (program components are delivered as planned by program developers) 
2) Exposure (frequency and dosage of implementation) 3) Quality of the delivery (how a teacher 
implements a program, e.g., enthusiasm, supervisor support) 4) Participants’ responsiveness 
(students’ responses to the intervention - e.g., the degree of their engagement, level of activity) 
and 5) Program differentiation (a manipulation check to make sure students in each condition 
receive only planned interventions). The authors recommended measuring all five aspects of 
fidelity. 



Purpose/objective/research question/focus of study: 

Description of what the research focused on and why. 

The purpose of this paper is to propose a framework for conceptualizing and assessing 
fidelity of implementation (FOI) in the context of a cluster-randomized design study. A research- 
validated mathematics curriculum for preschool children called Numbers Plus is used as an 
example to describe the role of curriculum implementation fidelity in the evaluation of programs, 
curricula, initiatives, and professional development activities. 
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Setting: 

Specific description of where the research took place. 



This three-year project was implemented in two states - Michigan and Indiana. 

Population/Participants/Subjects: 

Description of participants in the study: who (or what) how many, key features (or characteristics). 

A total of 17 partner programs participated in the project. They were a mix of Head 
Start, private child care, and faith-based programs (see Table 1). Data from the second year of 
the project came from a total of 37 preschool sites, including 149 teachers from 70 classrooms 
and 417 children (210 in the treatment group and 207 in the control group). Among the sites 
51.4% were in urban areas, 25.7% suburban, and 22.9% were rural area sites. 

Teachers’ ages ranged from 21 to 67 years. Most teachers were Caucasian (about 71%) 
and female (98%). Study teachers reported having been teaching anywhere from 0 to 30 years 
(M= 1 1.23). In addition, 57.1% of lead teachers in the treatment group and 64.7% of the lead 
teachers in the delayed treatment group had their highest degree in early childhood education or 
child development. 



Intervention/Program/Practice: 

Specific description of the intervention, including what it was, how it was administered, and its duration. 

Numbers Plus teaching strategies draw on the five underlying principles of active, 
participatory learning (Hohmann, Weikart, & Epstein Educating Young Children Third Edition, 
2008). The strategies include materials, manipulation, choice, child language and thought, and 
adult scaffolding. Together they describe a set of teaching practices that specify what teachers 
say and do, describe the types of learning opportunities teachers provide to children, and 
encourage teachers to use observation based information to identify children’s developmental 
levels. The five strategies align with and support the five process standards identified by NCTM 
- problem-solving, reasoning, communicating, connecting, and representing. 

A unique feature of Numbers Plus is that it sequences within rather than across activities. 
That is, instead of being introduced in a specific order (making it sometimes necessary to split 
children into ability groups), Numbers Plus activities assume that children of varying levels 
participate together in an activity and, with a few exceptions, no one activity must be mastered 
before another. To put this approach into practice, every activity provides teachers with 
examples of what children on a continuum of three levels (Earlier, Middle, and Later) may say 
and do. Correspondingly, for each level, the activity lists strategies teachers can use to support 
and extend children’s learning. 

In order to support this approach, the curriculum uses Key Developmental Indicators 
(KDIs) in each mathematical content area and topic. KDIs represent the important mathematical 
skills or concepts young children should acquire, or make substantial progress toward acquiring, 
by the end of preschool. Additionally, ten days of combined face-to-face and online, in-depth 
professional development was provided to teachers. The focus of that training was active 
learning, understanding math content, specific teaching strategies, and mastery of the small 
group time activity structure and process. 

Both delayed treatment control and experimental classrooms implemented the High/Scope 
curriculum in their classrooms as their overall curriculum model. During the school year, 
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experimental classrooms also implemented the Numbers Plus curriculum. The curriculum 
materials and activities primarily focused on content specific small group time activities. 
Follow-up ideas extend learning into other parts of the daily routine (such as meal times, during 
outside play, while children make plans and choices, at transitions, and so on) and in all areas of 
the classroom. 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimented design, secondary analysis, analytic 
essay, randomized field trial). 

The development and evaluation of this work has been a three year process. Year 1 
consisted of curriculum development and a pilot testing. Years 2 and 3 have consisted of 
implementation and evaluation. Data collection was done within a cluster randomized three- 
level design with randomization occurring at the site level (stratified on the type of program 
(Head Start and non- Head Start) and the number of classrooms per site (1-2 classrooms versus 
3-5 classrooms per site)). 



Data Collection and Analysis: 

Description of plan for collecting and analyzing data, including description of data. 

Data has been collected from teacher reports, child assessments, parent reports, and 
classroom observations. Because a central concern of the work has been insuring high quality 
implementation, tools were developed to measure implementation fidelity. 

The fidelity measures used in the study were based upon the curriculum theory and 
included multiple methods of assessment at multiple levels. The data on small-group time 
implementation was collected by administering the Mathematics Program Assessment, Part 2. 
The data on math throughout the day strategies was collected by using the Mathematics Program 
Assessment, Part 1. Activity Logs, teacher and director surveys, tracking training participation, 
and evaluating training were additional methods to assess the five aspects of fidelity. 

The development of the composite fidelity indices as well as issues related to weighting, 
aggregating the indices, standardizing, and developing a normative framework will be broadly 
described. Some of the indices (i.e., achieved relative strength that is based on Hedges’ g) 
created for the Numbers Plus study were modeled after the work done by Cordray and his 
colleagues (Cordray, Hulleman, and Lesnick, 2008) and some were specific to the evaluated 
curriculum. Multi-level modeling techniques also affect how FOI is measured: the need to 
balance the number of fidelity indices with sample size, correcting for clustering in the 
classrooms will be presented using the Numbers Plus data. Currently, each of the developed 
indices and different weighting schemes (i.e., when each component contributes equally as 
opposed to differential component weighting) are being explored to see how they function with 
multi-level data. 

Currently, the fidelity indices are being tested in mediator models. We will present on the 
relationship between FOI (utilizing various types of indices) and teacher- (e.g., knowledge, 
beliefs about math instruction, frequency of classroom practices, confidence in teaching about 
math, and anxiety related to teaching mathematics) and child-level outcomes (e.g., TEMA-3, 
Woodcock-Johnson III Applied Problems subtest, Child Observation Record) when controlling 
for covariates. 
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Findings/Results: 

Description of main findings with specific details. 

We argue that using various measures of fidelity of implementation provides stronger 
validity evidence and identifies mechanisms of how implementation works. In addition, the 
results of the analyses on student and teacher characteristics predicting FOI will be shared. 

Conclusions: 

Description of conclusions and recommendations of author(s) based on findings and over study. (To support the 
theme of 2009 conference, authors are asked to describe how their conclusions and recommendations might inform 
one or more of the above noted decisions — curriculum, teaching and teaching quality, school organization, and 
education policy.) 

FOI is an important concept in educational research, especially, studies implemented in 
schools. Well-designed studies assess FOI as a necessary step. It is, however, still true that there 
is little guidance on what aspects of fidelity to measure and how such assessment differs 
depending on the area of research, specific goals (development, efficacy, replication, or scale 
up), or a type of intervention used. This work will help researchers think critically about how to 
develop a framework of how to measure FOI and set guidelines that inform researchers, teachers, 
and policy makers about what curriculum or programs work for what students and under what 
conditions is proposed. 
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Appendixes 

Not included in page count. 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1 

Year 2 Percentage of Sites by Group Stratified by Type of Program and Number of Classrooms 
per Site 



Types of Sites 


Treatment 


Delayed 

Treatment 


Head Start 


60 


63.2 


Non-Head Start 


40 


36.8 


Smaller sites (1-2 classrooms per site) 


65 


68.4 


Larger sites (3-5 classrooms per site) 


35 


31.6 
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